We present a novel approach for tracking multiple people in video. Unlike past approaches which employ 2D representations, we focus on using 3D representations of people, located in three-dimensional space. To this end, we develop a method, Human Mesh and Appearance Recovery (HMAR) which in addition to extracting the 3D geometry of the person as a SMPL mesh, also extracts appearance as a texture map on the triangles of the mesh. This serves as a 3D representation for appearance that is robust to viewpoint and pose changes. Given a video clip, we first detect bounding boxes corresponding to people, and for each one, we extract 3D appearance, pose, and location information using HMAR.

In this supplementary document we provide additional information that were not included in the main manuscript due to space constraints. This includes details about a) the method, b) the implementation, c) the evaluation, and d) the visualization.

We present a novel approach for tracking multiple people in video. Unlike past approaches which employ 2D representations, we focus on using 3D representations of people, located in three-dimensional space. To this end, we develop a method, Human Mesh and Appearance Recovery (HMAR) which in addition to extracting the 3D geometry of the person as a SMPL mesh, also extracts appearance as a texture map on the triangles of the mesh. This serves as a 3D representation for appearance that is robust to viewpoint and pose changes. Given a video clip, we first detect bounding boxes corresponding to people, and for each one, we extract 3D appearance, pose, and location information using HMAR.

DeepSort is a machine learning model for tracking people, assigning IDs to each person. Traditionally, tracking has used an algorithm called Sort (Simple Online and Realtime Tracking), which uses the Kalman filter. Using the bounding boxes detected by YOLO v3, we can assign an ID and track a person by mapping bounding boxes of similar size and similar motion in previous and following frame. However, Sort presents the limitation that if a person hid behind an object and then reappeared, it is assigned a different ID. DeepSort solves this problem by using an AI model that compares similarity between people, thus reducing the issue of switching people's identities.